Papers with Neural language models
Scaling in Cognitive Modelling: a Multilingual Approach to Human Reading Times (2023.acl-short)
Copied to clipboard
| Challenge: | Neural language models provide conditional probability distributions over the lexicon that are predictive of human processing times. |
| Approach: | They propose to use a transformer-based model to generate probabilistic estimates that are less predictive of early eye-tracking measurements reflecting lexical access and early semantic integration. |
| Outcome: | The proposed models show that larger models capture late eye-tracking measurements that reflect the full integration of a word into the current language context. |
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural language models encode rich knowledge about entities and their relationships but common properties of nouns are difficult to extract because they are rarely explicitly stated in texts. |
| Approach: | They propose to extract perceptual properties from images and use them in an ensemble model to complement the information extracted from language models. |
| Outcome: | The proposed model improves noun property prediction compared to powerful text-based language models. |
Improved Natural Language Generation via Loss Truncation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing training procedures for neural language models seek to match the underlying distribution, leading to models that replicate unwanted behaviors such as hallucination during generation. |
| Approach: | They propose a procedure which optimizes for distinguishability by removing high log loss examples from a corpus to minimize the log loss. |
| Outcome: | Empirically, the proposed procedure outperforms baselines on distinguishability on a summarization task and shows that the models outperformed human references on the summarizing task. |
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior art aims to uncover meaningful properties within model representations, but it is unclear how faithfully such probes portray information that the models actually use. |
| Approach: | They propose a technique for generating counterfactual embeddings within models . they produce evidence that some models use a tree-distancelike representation of syntax . |
| Outcome: | The proposed technique produces evidence that some models use tree-distancelike representations of syntax in downstream prediction tasks. |
Does Vision Accelerate Hierarchical Generalization in Neural Language Learners? (2025.coling-main)
Copied to clipboard
| Challenge: | Neural language models (LMs) are arguably less data-efficient than humans from a language acquisition perspective. |
| Approach: | They investigate the advantage of grounded language acquisition over visual input to improve syntactic generalization. |
| Outcome: | The proposed model is less efficient than humans in language acquisition . it shows that visual input helps syntactic generalization, but not vision . |
You should evaluate your language model on marginal likelihood over tokenisations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural language models typically tokenise input text into sub-word units to achieve an open vocabulary. |
| Approach: | They propose that language models should be evaluated on their marginal likelihood over tokenisations instead. |
| Outcome: | The proposed approach is unsatisfactory and may bottleneck model out-of-domain performance. |
Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study (D19-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of neural models to compute and employ phrase-level features attached to a set of words, such as subject number or whquestion words. |
| Approach: | They examine whether models can represent constituent-level features, using coordinated noun phrases as a case study. |
| Outcome: | The proposed model can combine gender and gender features to drive downstream expectations, while having less success with gender agreement. |
Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Past work has found that query–probe disagreements occur in situations where either probes or queries are uncertain. |
| Approach: | They propose to identify three different classes of disagreement, which they term confabulation, deception, and heterogeneity. |
| Outcome: | The proposed methods are more accurate than LM outputs and better on uncertain answers than queries and probes. |
Privacy Regularization: Joint Privacy-Utility Optimization in LanguageModels (2021.naacl-main)
Copied to clipboard
Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg-Kirkpatrick, Robert Sim
| Challenge: | Neural language models have a high capacity for memorization of training samples . however, this can cause privacy degradation and disparate impact on subgroups of users . |
| Approach: | They propose two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy. |
| Outcome: | The proposed methods have favorable utility-privacy trade-off, faster training and uniform treatment of under-represented subgroups. |
An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Neural language models are often trained on positive examples, but recent studies suggest they are not robust enough to handle complex syntactic constructions. |
| Approach: | They propose to use negative examples to boost models' robustness on English sentences with a negligible loss of perplexity. |
| Outcome: | The proposed model is robust to negative examples in English with negligible loss of perplexity . |
The Importance of Generation Order in Language Modeling (D18-1)
Copied to clipboard
| Challenge: | Neural language models are universally autoregressive, generating sentences one token at a time from left to right. |
| Approach: | They propose a two-pass language model that generates partially-filled sentences and fills in missing tokens. |
| Outcome: | The proposed model produces partially-filled sentences and fills in missing tokens. |
Word Frequency Does Not Predict Grammatical Knowledge in Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural language models learn the grammatical properties of natural languages to varying degrees of accuracy. |
| Approach: | They focus on subject-verb agreement and reflexive anaphora to investigate whether there are systematic sources of variation in the language models’ accuracy. |
| Outcome: | The proposed model can learn grammatical properties from training data. |
Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model (2023.acl-long)
Copied to clipboard
| Challenge: | Existing interpretation methods fail to obtain faithful attributions on these models, thereby failing to reveal potential flaws and biases. |
| Approach: | They propose a Contrastive learning regularization method which calibrates the sentence representation of out-of-distribution examples and utilizes adversarial examples to introduce direction information in regularization. |
| Outcome: | The proposed method alleviates the model pathology while impacting generalization ability on in-distribution examples and thus helps interpretation methods obtain more faithful results. |
Baked-in State Probing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows language models trained on form can capture aspects of meaning without explicit state supervision. |
| Approach: | They propose to use probing to "bake" state knowledge into language models . they propose to probe for underlying world state knowledge via text prompts . |
| Outcome: | The proposed methods show that language models trained on form can capture the world state without state supervision. |
Experimenting with Power Divergences for Language Modeling (D19-1)
Copied to clipboard
| Challenge: | Language models are an important component in many NLP tasks, where they provide prior knowledge on the language used. |
| Approach: | They propose to use power divergences to prioritize learning on frequent or rare words . they use a sample-based objective to approximate a softmax and noise-constrained estimate . |
| Outcome: | The proposed power divergences can be used to prioritize learning on the frequent or rare words and lead to general performance improvements. |
Similarizing the Influence of Words with Contrastive Learning to Defend Word-level Adversarial Text Attack (2023.findings-acl)
Copied to clipboard
| Challenge: | Neural language models are vulnerable to word-level adversarial text attacks . previous word-based search methods assume important words influence prediction . |
| Approach: | They propose a method for similarizing the influence of words with contrast learning that encourages model to learn sentence representations in which words of varying importance have a more uniform influence on prediction. |
| Outcome: | The proposed method is compatible with various training methods and improves model robustness against various adversarial attacks. |
Softmax Bottleneck Makes Language Models Unable to Represent Multi-mode Word Distributions (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language models can't predict the probability distribution over next words or masked words because of a single hidden state. |
| Approach: | They propose a multi-facet softmax to address the limitations of softmax . they propose LMs that can predict the next word distribution better as their sizes grow . |
| Outcome: | The proposed model achieves two-fold improvements in the perplexity of GPT-2 and BERT. |
CAPE: Context-Aware Private Embeddings for Private Language Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to obtain text representations or embeddings with these models encoding personally identifiable information may lead to privacy leaks. |
| Approach: | They propose a novel approach which combines differential privacy and adversarial learning to preserve privacy during training of embeddings. |
| Outcome: | The proposed approach reduces private information leakage by 3% over the current method. |
How Gender Interacts with Political Values: A Case Study on Czech BERT Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural language models are trained on large text corpora that contain value-burdened content and often capture undesirable biases, which the models reflect. |
| Approach: | They propose a method to measure the model's perceived political values by comparing Czech with a representative value survey. |
| Outcome: | The proposed method does not assign statement probability following value-driven reasoning and there is no systematic difference between feminine and masculine sentences. |
Improving Text Generation with Student-Forcing Optimal Transport (2020.emnlp-main)
Copied to clipboard
Jianqiao Li, Chunyuan Li, Guoyin Wang, Hao Fu, Yuhchen Lin, Liqun Chen, Yizhe Zhang, Chenyang Tao, Ruiyi Zhang, Wenlin Wang, Dinghan Shen, Qian Yang, Lawrence Carin
| Challenge: | Maximum likelihood estimation (MLE) is used to train models, but during testing, the model is conditioned on previously generated tokens, resulting in exposure bias. |
| Approach: | They propose to use optimal transport to match the sequences generated in MLE and test modes to reduce exposure bias. |
| Outcome: | The proposed method is validated on machine translation, text summarization, and text generation tasks. |
JCoLA: Japanese Corpus of Linguistic Acceptability (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural language models have exhibited outstanding performance in downstream tasks, yet there is limited understanding regarding the extent of their internalization of syntactic knowledge. |
| Approach: | They introduce a dataset that analyzes sentences annotated with binary acceptability judgments from linguistic textbooks and handbooks and splits them into in-domain and out-of-domain data. |
| Outcome: | The proposed datasets show that models can surpass human performance for in-domain data while no models can exceed human performance on out-of-domain datasets. |
Rethinking Word-level Adversarial Attack: The Trade-off between Efficiency, Effectiveness, and Imperceptibility (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural language models have demonstrated impressive performance but remain vulnerable to word-level adversarial attacks. |
| Approach: | They propose two standardized search spaces to address the problem of word-level adversarial attacks. |
| Outcome: | The proposed search spaces improve performance and trade-offs in different scenarios. |
Targeted Syntactic Evaluation on the Chomsky Hierarchy (2024.lrec-main)
Copied to clipboard
| Challenge: | a novel evaluation paradigm for targeted syntactic evaluations is proposed . we create formal languages that abstract four syntaktic phenomena in natural languages . |
| Approach: | They propose a new evaluation paradigm for Targeted Syntactic Evaluations . they create formal languages that abstract syntactical phenomena in natural languages . |
| Outcome: | The proposed evaluation paradigm evaluates language models on language modeling tasks . it shows that they can capture the structural patterns of the (Adj)n NP type formal language . |